Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add AREA_CODE high performance geocoding #208

Merged
merged 2 commits into from
Feb 5, 2025

Conversation

joaquimds
Copy link

@joaquimds joaquimds commented Feb 5, 2025

Part of the work required for the data pipeline V2: https://linear.app/commonknowledge/issue/MAP-973/planning-for-data-import-pipeline-v2

This adds a new Geocoder: AREA_CODE_GEOCODER_V2. To enable it, the geocoding config must have only one element, with the "type" set to "area_code".

This Geocoder uses dataloaders to ensure maximum parallelisation.

To Test

Add an AreaData source, where each row is identified to an area by code. Set up the geocoding_config to look something like:

  "type": "AREA",
  "components": [
    {
      "type": "area_code",
      "field": "oa21cd",
      "value": "",
      "metadata": {
        "lih_area_type__code": "OA21"
      }
    }
  ]
}```

Then run the import on the data source.

To enable this, the geocoding config must have only one element,
with the "type" set to "area_code".
@commonknowledge-bot commonknowledge-bot deployed to feat/area-code-geocoding - meep-database PR #208 February 5, 2025 13:22 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-backend PR #208 February 5, 2025 13:24 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-backend PR #208 February 5, 2025 13:24 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-frontend PR #208 February 5, 2025 13:24 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot deployed to feat/area-code-geocoding - meep-intelligence-hub-worker PR #208 February 5, 2025 13:24 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-frontend PR #208 February 5, 2025 13:24 — with Render Destroyed
Copy link
Member

@janbaykara janbaykara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for adding tests for this.

Comment on lines 23 to 24
filters: dict = {}
select_related: list = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be = dict() and = list() for immutability's sake?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think changing them to dict() and list() would fix the immutability issue, but I think simply removing the default values works – the factory sets them to fresh {} and [] on class creation.

@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-worker PR #208 February 5, 2025 13:43 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to feat/area-code-geocoding - meep-intelligence-hub-backend PR #208 February 5, 2025 13:43 — with Render Destroyed
@joaquimds joaquimds merged commit 5154e33 into main Feb 5, 2025
4 checks passed
@joaquimds joaquimds deleted the feat/area-code-geocoding branch February 5, 2025 13:54
Copy link

sentry-io bot commented Feb 6, 2025

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ OperationalError: the connection is closed hub.tasks in wrapper View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants